Data Quality: An Attribute Framework for Large Systems

In previous work we examined a process for cleaning data from a single source. We proposed a framework with three attribute categories— critical, logical, irrelevant— that enables users to interpret the effect of dirty data on operational decisions. To extend the framework to large systems, we suggest adding a fourth category “used” as a set of tags. The metadata on the tags describes the usage— how users modified the attributes to meet their requirements. One database we examined contains data feeds multiple sources within an international firm plus external feeds containing financial market information. As the data flows through the corporation to the users, additional attributes are added and multiple databases are integrated. At each stage users endeavor to maintain clean data for attributes important to their function. Downstream analysts, however, do not know which attributes have been used. This paper describes the framework extension and the metadata tags.